Systems and Methods to Define and Monitor a Scenario of Conditions

ABSTRACT

Systems and methods to define a scenario of conditions comprising the steps of defining at least one condition for at least one educational objective, the at least one condition being represented by a constraint and scheduling the conditions into a scenario of conditions. In some embodiments, the scheduling is performed by analyzing the constraints using constraint programming. In some embodiments, the constraints comprise mathematical or computational constraints representing a range of variables. Also disclosed are systems and methods to monitor a scenario of conditions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/502,120, filed on Apr. 13, 2012 and entitled “SYSTEMS AND METHODS TODETERMINE AND MONITOR A SCENARIO OF CONDITIONS”; U.S. patent applicationSer. No. 13/502,120 is a U.S. National Stage application ofInternational Application No. PCT/US2010/055525, filed on Nov. 4, 2010and entitled “SYSTEMS AND METHODS TO DEFINE AND MONITOR A SCENARIO OFCONDITIONS”; International Application No. PCT/US2010/055525 claims thebenefit of U.S. Patent Application Ser. No. 61/258,236, filed on 5 Nov.2009 and entitled “SYSTEMS AND METHODS TO DEFINE EDUCATIONAL SCENARIOS”;and the entire contents of each are is herein incorporated by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Contract#N61339-08-C-0017 awarded by the U.S. Navy. The Government has certainrights in the invention.

REFERENCE TO SEQUENCE LISTING, A TABLE, OR A COMPUTER PROGRAM LISTINGCOMPACT DISC APPENDIX

Not Applicable.

BACKGROUND OF THE INVENTION Field of the Invention

This invention relates to simulated environments, in particular tosimulation environments that utilize scenarios.

Description of the Prior Art

Immersive scenarios, such as those found in military exercises andcomputer-based games, are often viewed as a sequence of discretepartially scripted events. In military exercises, these may be found ina planning document called the Master Scenario Events List (MSEL). Eachscripted MSEL event involves specific conditions and actions by specificlive, virtual, or constructive entities, and the conditions and actionstake place at specific virtual locations at specific scenario times.Computer-based games are scripted similarly.

Current scenario definition languages have been designed to provide highfidelity detail but generally fail to define clearly the linkagesbetween scenario events and training objectives.

BRIEF SUMMARY OF THE INVENTION

The following summary is included only to introduce some conceptsdiscussed in the Detailed Description below. This summary is notcomprehensive and is not intended to delineate the scope of protectablesubject matter, which is set forth by the claims presented at the end.

It is an object of one embodiment of the systems and methods todetermine and monitor a scenario of conditions to provide a way to viewscenarios comprising a partially ordered collection of partiallyspecified conditions and actions by relaxing these scenario conditions(when possible) in order to gain flexibility in sequencing, scheduling,positioning, monitoring and replaying scenario events or conditions. Insome embodiments, the scenario defining and monitoring system includes:a pedagogically sound scenario definition language; a scenario authoringtool; a scenario analysis tool to determine the feasibility ofscenarios; and a decision support tool for trainers, systems oroperators to monitor performance, give condition satisfaction status andprovide corrective guidance while encountering scenario conditions orevents such as a group of conditions or a series of conditions.

It is an object of one embodiment of the invention to provide a methodof defining a scenario of conditions, said method comprising the stepsof defining a plurality of conditions for at least one objective, theplurality of conditions being represented by at least one constraint andscheduling the conditions into a scenario of conditions. In someembodiments, the at least one constraint is a mathematical expressionrepresenting at least one variable, the step of scheduling the pluralityof conditions into a scenario of conditions comprises optimizing theobjective using constraint programming or the objective is aneducational objective. The step of scheduling the plurality ofconditions into a scenario of conditions can also comprise satisfyingthe objective using constraint programming.

It is an object of other embodiments of the invention to provide amethod of defining a scenario of conditions wherein a value of the atleast one variable is defined by an entity, a resource or the value is apriority.

It is an object of one embodiment of the invention to provide a methodof monitoring a user on a scenario of conditions comprising presenting ascenario of conditions to a user, the scenario of conditions comprisinga plurality of conditions for at least one objective, the plurality ofconditions being represented by at least one constraint, monitoring anexecution of the user of at least one condition, creating at least oneactual condition of the scenario of conditions based on the execution ofthe user, and communicating the measure actual condition. In someembodiments, the constraint is a mathematical expression that representsa range of variables, the objective is an educational objective or oneof the conditions is one of a plurality of 3D geographic locations of anentity.

It is another object of some embodiments to provide a method ofmonitoring a user on a scenario of conditions wherein the scenario ischanged dynamically during the execution of the user based on the actualcondition, the scenario of conditions is changed dynamically during theexecution of the user based on a second actual condition of an executionof a second user or the scenario of conditions is changed by optimizinga plurality of the constraints using constraint programming.

It is an object of one embodiment of the invention to provide a computerbased system for defining a scenario of conditions comprising a means toreceive a plurality of conditions for at least one objective, theplurality of conditions being represented by at least one constraint anda means to automatically schedule the conditions into a scenario ofconditions. In some embodiments, the constraint is a mathematicalexpression and the means to automatically schedule the conditions into ascenario of conditions comprises optimizing the constraints usingconstraint programming. In some embodiments, the objective is aneducational objective.

It is another object of embodiments of the invention to provide a methodof monitoring a user on a training scenario comprising the steps ofpresenting a training scenario to a user where the training scenariocomprising at least one condition for at least one training objectiveand the at least one condition being represented by a constraint,monitoring a user's performance following the training scenario,creating condition satisfaction measures of the user's performanceagainst the conditions and comparing the condition satisfaction measuresto the conditions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In order that the manner in which the above-recited and other advantagesand features of the invention are obtained, a more particulardescription of the invention briefly described above will be rendered byreference to specific embodiments thereof which are illustrated in theappended drawings. Understanding that these drawings depict only typicalembodiments of the invention and are not therefore to be considered tobe limiting of its scope, the invention will be described and explainedwith additional specificity and detail through the use of theaccompanying drawings in which:

FIG. 1A illustrates a conceptual comparison of a conventional scenariocompared to a training scenario defined by embodiments of the disclosedmethods;

FIG. 1B illustrates a general flow chart of one embodiment of methods todefine a scenario of conditions with unknowns;

FIG. 1C illustrates a general flow chart of one embodiment of methods todefine a scenario of conditions with unknowns identified;

FIG. 2 illustrates one embodiment of the top-level elements of HumanPerformance Markup Language (HPML) used by PRESTO;

FIG. 3 illustrates one embodiment of the Entity elements used by PRESTO;

FIG. 4 illustrates one embodiment of the Scenario element used byPRESTO;

FIG. 5 illustrates one embodiment of the Training Objective Instanceselement used by PRESTO;

FIG. 6 illustrates one embodiment of the Library element used by PRESTO;

FIG. 7 illustrates one embodiment of a training objective template and atraining objective example;

FIG. 8 illustrates one embodiment of a training scenario schedule;

FIG. 9 illustrates one example of an analysis process;

FIG. 10 illustrates a screen shot of one embodiment of a monitoringtool;

FIG. 11 illustrates a screen shot of one embodiment of a conditioncapture tool;

FIG. 12 illustrates an example of a sequence diagram of conditionmonitoring;

FIG. 13 illustrates an example use case scenario;

FIG. 14 illustrates an example of use cases for a training objectiveauthor;

FIG. 15 illustrates an example of use cases for a scenario designer;

FIG. 16 illustrates another example of use cases for a scenariodesigner;

FIG. 17 illustrates an example of use cases for a trainer;

FIG. 18 illustrates an example of use cases for a Semi-Automated Forces(SAF) operator;

FIG. 19 illustrates a functional diagram of one embodiment of Use Casesof a scenario defining system;

FIG. 20 illustrates a functional overview of one embodiment ofconstraint programming logic;

FIG. 21 illustrates a component diagram of one embodiment of a scenariodefining system; and

FIG. 22 illustrates a component diagram of one embodiment of a scenariomonitoring system.

DETAILED DESCRIPTION OF THE INVENTION

Methods of defining a scenario of conditions, such as educational andtraining scenarios, as well as computer based scenario defining systemswill now be described in detail with reference to the accompanyingdrawings. It will be appreciated that while the following descriptionfocuses on methods and systems that define and monitor educational andtraining scenarios, the systems and methods disclosed herein have wideapplicability. For example, the methods and systems described herein maybe readily employed with project planning or task/resource allocation aswell as any educational system such a specifically programmed computerbased learning system. Notwithstanding the specific example embodimentsset forth below, all such variations and modifications that would beenvisioned by one of ordinary skill in the art are intended to fallwithin the scope of this disclosure.

Current scenario definition methods provide detail for discrete,precisely specified events or conditions planned to occur during thescenario but generally fail to define clearly the linkages betweenscenario events or conditions and training objectives. This isparticularly true for high-fidelity scenarios. One embodiment of thescenario defining and monitoring systems, termed PRESTO (PedagogicallyRelevant Engineering of Scenarios for Training Objectives) as usedherein is based on an alternate way to view scenarios. In this view,scenarios consist of a partially ordered collection of partiallyspecified conditions and actions by live, virtual, or constructiveentities, and the conditions and actions take place with partiallyspecified location and timing constraints. In other words, PRESTOtechnology exploits the relaxation of scenario specifics (when possible)in order to gain flexibility in sequencing, scheduling, positioning andreplaying scenario events or conditions.

In particular, the PRESTO embodiment includes: a pedagogically soundscenario definition language; a scenario authoring tool; a scenarioanalysis tool to determine the feasibility of scenarios; and a decisionsupport tool for trainers, systems or operators to monitor performance,give condition satisfaction status and provide corrective guidance whileencountering scenario conditions or events such as a group or series ofconditions.

FIG. 1A illustrates some of these differences. In conventional, MasterScenario Events List (MSEL) based scenarios, the ovals show a plannedset of highly specific scenario events. In PRESTO-based scenarios, someevents are equally specific, but some are not. The relaxed eventspecifications are shown by the different sized ovals. In theconventional scenario, circumstances can often make it impossible forthe actual sequence of events to match the planned sequence of events,but PRESTO-based scenarios provide wider latitude for accommodating theunexpected. For example, suppose a MSEL-based scenario requires that atrainee be at a location 223.656, 194.331 at time t=32.54. APRESTO-based scenario might relax those conditions and simply requirethat the trainee be within 1 mile of that position at some time aftert=10 but before t=75. If unexpected events make it impossible to meetthe MSEL-based requirements, there is still a chance that thePRESTO-based requirements can be met, because they are considerably lessdemanding although still pedagogically rigorous.

By using mathematical or other computational expressions to modelaspects of scenarios and conditions, tools such as constraintprogramming can be used as a means to automatically define thescenarios. For example, training scenarios including trainingobjectives, conditions or preconditions of training objectives, can bemodeled with tools such as constraint programming to schedule trainingactivities. Constraint programming can be used as a means to solveconstraint satisfaction problems such as defining scenarios to satisfyall the conditions. Constraint programming can also be a means to solveconstraint optimization problems such as defining the optimal scenarioof conditions. In addition, the use of mathematical expressions asmodels provides helpful tools that can be used to monitor, manage andoptimize the progress of a user through the training scenario.

This approach differs from earlier work in constraint programming inthat it is focused on attempting to schedule conditions that canoptimize a set of events such as working on training objectives oradvancing the plot line, and not on maximizing the throughput of asystem, as would be the case for job-shop or workflow applications. Inparticular, the application of constraint programming to training,educational, and entertainment-based situations is unique in that thefocus is on optimizing events rather than on optimizing the performanceof the system. Here, the nature of the optimization—the objectivefunction—could be address any of a multiplicity of things. For example,with this approach it is possible to address a variety of goals such asminimizing the length of the scenario, maximizing the number of trainingobjectives, maximizing a certain number of conditions required for eachevent, maximizing the total value of all events given a value for eachevent, and a large number of other sensible goals. As used above, andthroughout this description, an event can be one or more conditionswithin a scenario.

One Embodiment of Methods to Define Training Scenarios:

One embodiment of methods to define scenarios is illustrated by thePRESTO system (also “PRESTO”) which focuses on the key conditions of thescenario experiences that must be met to provide the needed context fora trainee to train. With the conditions satisfied, other factors of theexperience can vary as needed to provide flexibility in the events ofthe scenario while still satisfying the training objectives. The PRESTOsystem can be used during scenario development to analyze a scenario forfeasibility and during the training exercise to support thetrainer/operator to bring about the conditions which will give thetrainees the most effective training experience. PRESTO allows trainersto create training objective focused scenarios more quickly thanconventional methods. The scenarios defined are more flexible inproviding the situations needed to achieve the training goals.

In general, the methods of the defining a scenario of conditions isshown in FIGS. 1B and 1C. As shown in FIG. 1B, the method embodiment 100comprises identifying at least one objective at 110, identify objectiveconditions at 120, mathematically representing the objective conditionsat 130 and defining a scenario of conditions with constraint programmingat 150. The result of this is a scenario of conditions using constraintprogramming that can be used to generate scenarios compliant withobjective. This embodiment represents objective conditions that havemathematical unknowns which can be represented by a range of unknownvalues.

FIG. 1C illustrates one embodiment of the methods of defining a scenarioof conditions 100 where the objective is a scenario objective withsub-objectives and the constraints are at least partially populated withunknown values. An example of this embodiment would be a scenarioobjective to train a group of trainees on several sub-objectives (e.g.tasks) and each of these tasks has their own conditions. The conditionsare mathematically represented and the unknowns within the conditionsare identified at 140. These unknowns (e.g. task priority) can populatethe conditions to define a scenario of conditions at 150 or the unknowns(e.g. actual task accomplishment) can populate the conditions andscenarios so that performance against the conditions can be monitored orrescheduled at 160.

As referred to in FIGS. 1B and 1C and throughout this description, acondition is any type of constraint or requirement used to define anobjective and help determine whether the objective is met. For exampleand not for limitation, a condition may be whether an entity is present,what skills an entity has, what speed the entity is traveling and towhat degree an objective has been met. By representing these types ofobjective conditions as mathematical constraints, it is possible torepresent the conditions more flexibly, such as in a mathematical range,that can more accurately represent methods to meet the objective or tobetter satisfy a group of objectives. As referred to in FIG. 1C andthroughout this description, an unknown is a value or otherrepresentation of a variable that is used within a condition orconstraint. For example and not for limitation, an unknown may be thepriority of a training objective, estimated resource values oridentification of an entity to be trained through a scenario as well astraits or properties of this trainee or the equipment they work with inthe scenario.

Embodiments of these methods can also include monitoring the actualscenario conditions to populate condition unknowns and determine whichobjectives have conditions fully satisfied (in the constraint range) orpartially satisfied (in the constraint range), and which plannedobjectives are no longer possible (outside of the constraint range)given actual events (other conditions and/or constraints) that havetranspired during the scenario. In these types of situations, conditionunknowns can further include performance data captured during an actualtraining session such as location, timing, condition satisfactionmeasures or any other information that can be used with the systemconditions and constraints.

Embodiments can also include modifying objective conditions to reflectvariable conditions or constraints such as the existing skills of onetrainee as compared to another trainee or varying priorities ofobjectives desired.

Embodiments can also include conditions for emergent objectives. Forexample, in the context of training objectives, emergent trainingobjectives are not planned, in the sense that the scenario would notattempt to bring them about, but that nevertheless occur during thecourse of actually executing the scenario. PRESTO can recognize such“teachable moments” and present alert scenario personnel such asinstructors that they have occurred.

PRESTO uses a number of data items to facilitate scenario planning,training objective management and optimization. Although HumanPerformance Markup Language (HPML), an XML-schema based language fordescribing human performance in simulators, is not necessary for allembodiments, it is useful, and it contains the major data structuresused in some embodiments of PRESTO. The top-level elements may be seenin FIG. 2: scenario entities; scenario element; instances of trainingobjectives (TO Instances) accommodated by the scenario; scenarioinformation system configuration; and a library of predefined trainingobjectives, conditions, resources and performance measurements that canbe used in the scenario. These top level elements are used to identifyand sometimes provide values for the objective conditions.

Scenario entities describe the entities in the scenario, as may be seenin FIG. 3. The entities provide values for condition unknowns. Forexample, with PRESTO, there are five elements that describe scenarioentities: Trainees—the individuals and teams being trained; Trainers—thepeople and systems doing the training. In addition to instructors, thiselement also includes simulators and SAF applications; Tasks—the work tobe assigned to the trainees; Roles—the roles that trainees or trainersmay take on in performing a task; Assignment—the pairing of an actor(trainee or trainer) with a task and role; and Areas—places wheretraining activity will take place which could specify a specificlocation, or a range of locations.

The scenario element holds any pre-existing descriptions of thescenario, independent of to-be-scheduled training objectives, as can beseen in FIG. 4. This information includes the instructions given toscenario participants as well as information that described anypre-existing conditions in the scenario such as a Master Scenario EventsList (MSEL).

The configuration element provides utility information used to connectPRESTO or other external systems to the available electronic data.Real-time data from the simulation environment provides PRESTO with thecurrent state of the simulation environment so that monitoring andanalysis can be done. Real-time data is provided periodically andcontains information such as the following: time stamp—the time the datawas observed; entity id—the identification of the entity that wasobserved; location—the location of the entity at the time it wasobserved; flight angles—pitch, roll and heading of the entity;velocity—the current speed of the entity; altitude—the current altitudeof the entity; and activities such as deploying a sensor or firing aweapon.

The library element contains information that can be applied to a widevariety of situations, and includes information such as trainingobjectives, performance measurements and assessments, resourcedefinitions, and constraints used by the PRESTO scheduler, as shown inFIG. 6. A training objective identifies the purpose of a specific pieceof training and includes a bundle of information defining conditions orrequirements that come into play when the trainee works on it, includingprerequisites, required resources, scenario preconditions, applicableperformance measurements and assessments, and, if necessary, subordinatetraining objectives that contain the same information about a portion ofthe overall training objective. The training objective described herecan be thought of as a template that describes all of the conditionsthat are necessary for a trainee to make progress on that trainingobjective. The training objective template contains “slots” whichrepresent unknowns in a constraint and usually can be represented asvariables in a mathematical or computational representation of theconstraint. The slots are filled in (if possible) with values before oras the PRESTO processing proceeds in order to make a training objectiveinstance have specific values.

For the PRESTO embodiment, training objectives instances are used todefine potential or actual training objectives populated with some orall of their unknowns. As the training objectives (also TOs) representspecific events or tasks to be trained and the conditions associatedwith specific training objectives, training objective instances (also TOInstances) represent partially or fully populated training objectiveswith data for the unknowns. Since the specific training objectives canbe instantiated differently or for multiple times for training exercisesor scenarios and for different entities, these training objectiveinstances are used to describe how the training objective is to be usedfor that instance (for example that trainee) in the scenario, as shownin FIG. 5. Example information describing the training objectiveinstance itself can be such information as the resources involved, thetrainees involved, the performance measures that will be used to measureand assess trainee performance, and any special scenario conditionsrequired by this particular instance.

When a trainee is assigned a training objective, additional data can beprovided to provide the unknowns that help describe the trainingobjectives a trainee should work on. For the PRESTO embodiment, thisdata includes information such as the priority and role for the trainee.The trainee training needs data consists of a list of data items withthe following content: trainee—an identifier of the trainee thistraining needs data applies to; training objective—the trainingobjective the trainee should work on; priority—a priority for thetrainee for this training; and objective role—the role in this trainingobjective that the trainee should experience.

The above data can be provided for each entity in the scenario (SAFs andtrainees). Other simulation environment data is also obtained such astime of day and weather conditions.

The above conditions and data can be represented by a mathematicalconstraint or other constraint. Most commonly, this is an assertioninvolving integers: an altitude is greater than a certain (integer)amount, the amount of fuel left is between two (integer) values, and soon. But there is nothing to prevent these constraints from havingreal-valued or symbolic elements or sets of values, except that certaincombinations can be more computationally demanding than others. Forexample, scenario entities can be represented by sets of capabilitieslike speed, weapons, and range; other scenario elements such as weathercan be represented by an enumeration such as <sunny, partially cloudy,cloudy, precipitating>; locations can be represented by specifying a 3dimensional distance around a point; airspeed and altitudes can berepresented by ranges; and so on.

One type of constraint is a precedence constraint. Precedenceconstraints specify that one or more events, conditions, or trainingobjectives must be preceded by one or more other events, conditions, ortraining objectives. The set of precedence constraints defines a partialordering of the events, conditions, or training objectives in the sensethat not all events are ordered with respect to each other. For example,it is important that a missile be fired (an event) before there is adamage assessment event, but this sequence could come either before orafter a weather forecast event.

As described below, the use of certain types of constraints allow theconditions and training objectives to be analyzed with tools such assoftware programs and constraint programming tools.

One goal of PRESTO is to accomplish scenario engineering to increasetraining opportunities. PRESTO accomplishes this by analyzing trainingobjectives in the context of a scenario either during scenario planningor during training exercise time. One of the processes that PRESTOperforms is to determine how to schedule, or “fit”, training objectivesinto either the scenario as it is being planned or into a trainingexercise as it is being conducted.

To illustrate some of the processes of PRESTO, consider the idea of atraining objective as a template (as described above with the trainingobjective). PRESTO processing accomplishes these major activities withrespect to training objectives:

-   -   instantiate training objectives for the trainee—This involves        taking the training objective template for the given training        objective and filling in all of the information that is known at        this point. We can call these potential training objectives        (PTO) because at this point we do not know if the training        objective conditions can be satisfied.    -   determine values for PTO “slots”—An example of a PTO slot is a        variable condition such as the resources needed for the training        objective. In the case of an attack situation PTO which requires        a target and two aircraft in the area, each of these three        resource slots represent a potentially unknown variable of a        constraint to be later filled in with a specific trainee or SAF        value to satisfy the resource conditions for the PTO.    -   optimize—The optimization occurs once the values for the PTO        slots are determined or estimated. This is handled by a        constraint programming (CP) components of PRESTO and the set of        algorithms available in the CP system used by PRESTO.

Because these objectives or conditions can be represented bymathematical constraints, they can be analyzed and modeled using toolssuch as CP.

To illustrate the above process, consider the training objective examplefor buddy lasing (a kind of fighter plane attack) shown in FIG. 7. Thecontents of a training objective template is shows in the left side andthe buddy lasing training example is shown on the right side of FIG. 7illustrating the conditions that must be satisfied and the slots thatmust be filled in with actual values in order for conditions to besatisfied and the training objective to be realized for the trainee. Theslots are the unknowns or variables in the conditions which requirevalues to be assigned to them to determine how they satisfy theconstraints or conditions. Given this example, the following can bedetermined before the trainee can work on this training objective:

-   -   A trainee with a buddy lasing training objective would fill the        “L” (for laser) resource slot, in this case the trainee is        called Bravo-1. A target (resource slot named “T”) and another        helicopter (resource slot named “M” who would fire a missile)        will also need to be identified to fill the other resource        slots. In the example T is assigned to the SAF called Ship-75        and M to another trainee called Bravo-2.    -   The context conditions will need to be checked against the        current conditions of the scenario to make sure they can be met.        These conditions have other implications such as Over water        which will depend on the location of L, T and M and Sun angle        which will depend on the time this training objective is        scheduled.    -   Timing conditions need to be checked. In this case L cannot work        on this training objective unless they have just completed a        “Communicate with MOM” training objective. When the PRESTO        method is able to schedule this training objective the Start and        End time slots are filled in.    -   The geometric and spatial conditions must be checked. In this        example, these conditions define a geometric configuration        between the lasing helicopter (“L”), the target (“T”) and the        missile firing helicopter (“M”). There is a distance range        (“D_(L)” and “D_(M)”) specified between each of the entities and        also an angle (“A”). For scenario planning purposes these        conditions are used to determine where to move entities or where        to place them (for example, where to place the target so that        the geometry is correct). For scenario monitoring purposes, when        the training scenario is running, these conditions determine the        pre-conditions for progress on the training objective to        commence.

During the running of the scenario, the location slots can be filled inwith the actual locations of the entities for the training objective andthe geometrical conditions checked for satisfaction.

The challenge of scheduling the training objective instances is thatmany of them are dependent on each other. So, a brute force approach isto just try all combinations of values for each of potential time value.This is computationally difficult. To address the complexity associatedwith trying to complete the training objective templates, the PRESTOsystem converts the training objective conditions to temporal, spatial,spatiotemporal, resource and other kinds of mathematical or othercomputational constraints. These mathematical constraints can take avariety of forms, ranging from MSEL-like specific locations to veryloose regional specifications to inter-training objective timingdependencies to weather and sea state requirements. For any one trainingobjective for any one trainee, it is often the case that bringing aboutthe required conditions (that is, satisfying the constraints) is easyeven without automation, but it is also often the case that satisfyingthe constraints that represent the combination of training objectiveconditions for all trainees is extremely difficult. In fact, it issometimes the case that there is no solution that satisfies all theconstraints. In this case, what is needed is a solution that satisfiesas many high-priority conditions as possible. When there is a solution,we have satisfied the constraints; when there is not, the solution isthe optimal compromise.

Within PRESTO, each training objective condition is expressed formally,and all of them are fed to a mathematical CP application that satisfiesor optimizes them. To complete the population of the training objectiveslots, the general way is to intelligently pick some of the values forthe slots and then use those values to limit the choices for the valuesof the other slots (based on the defined conditions or constraints).PRESTO uses a number of approaches which pick certain slot values firstand then optimize on other slot values.

One example of a suitable CP application into which these values are fedis the ECLiPSe Constraint Logic Programming environment. Another exampleis the COMET Constraint Programming environment. Both of theseenvironments implement a type of constraint-based local search, in whichan initial solution is found, and then a local search is performed formodifications that will be more optimal. A CP environment and theassociated solvers also provide numerical and logical methods toapproach the scheduling of activities in the scenario. This can be doneboth before the exercise begins in order to determine the best scenarioplan, and it is done several times during the exercise in order todetermine the best plan for the remaining time in the scenario, giventraining objectives actually achieved and the current state of thetraining exercise. The plan, also called a schedule or scenario, isexpressed as a Gantt chart, which shows the planned sequence ofactivities for the scenario, displayed by trainee, instructor, orentity. FIG. 8 shows a sample schedule from the PRESTO Scheduling Tool.

Regarding scheduling, FIG. 9 shows one approach of schedulingactivities. Specifically, this approach assigns entities to resourceslots in a training objective template based on the constraint of theirgeographical location. Then the approach attempts to generate scheduletimes that satisfy the timing conditions. This approach has limitationsin the situation when the entities have not been located yet (as in thescenario planning-time phase).

The schedule generation process shown in FIG. 9 has the following steps:generate clusters at step 972, generate feasible training objective listat step 974 and generate a schedule at step 976.

Step 972, generate clusters, takes an initial placement of all of theentities (trainees, SAFs and targets) and generates clusters ofentities. These clusters are the groups of entities that will be used tosatisfy training objectives. For example, consider a buddy lasingtraining objective. It would be most efficient to be able to buddy lasewith another entity that is physically near and hence would be includedin the cluster. The clustering process can be parameterized so that moreor fewer clusters can be determined or clusters can be generated forlarger regions. The clusters could also be aligned with organizationalunits. The results of this processing step are a set of clusters thatcontain a set of entities.

Step 974, generate feasible training objective list, analyzes eachcluster to generate a list of feasible training objectives for thetrainees in the cluster. First, a list of all of the training objectivesfor all of the trainees in the cluster is gathered. Then the resourcerequirements are checked to see if they can be satisfied. For example,if one trainee of the cluster has a fire missile training objectivewhich requires a SAF target, but there are no targets in the cluster,then this training objective is not feasible (this problem can bemitigated by manipulating the targets to satisfy this requirement). Theresult of this step is a set of training objectives for each clusterthat have feasible resource needs.

Step 976, generate schedule, generates a schedule for each cluster. Thetraining objectives for the cluster are scheduled taking into accountthe following items. The source information for this step is generatedby the PRESTO training objective authoring tool. Source information forthis step can include timing dependencies, resource contention, priorityand/or duration. Timing dependencies define training objectivedependencies on time such as, for example, TO-1 must occur before TO-2,or TO-3 must happen concurrently with TO-4. Resource contention defineresource constraints such as, for example, if two trainees need a targetfor a training objective and there is only one target available, thenthose two training objectives cannot be scheduled at the same time.Priority can define the priority of that training objective to thattrainee. The scheduling optimization algorithm can favor the higherpriority training objectives and schedule them earlier in the trainingexercise when possible. Duration can define the duration of the trainingobjective and the duration of the training objective can be constrainedto fit into the schedule.

The result of the scheduling process is a schedule of trainingobjectives for each trainee and a list of analysis and feasibilitymessages. These messages describe conditions that could not be satisfiedduring the scheduling process. For example, trainee “A” cannotaccomplish training objective “attack target” because not enough targetsare available.

One Embodiment of Method to Monitor Training Scenarios:

As expected, during the actual execution of a scenario by a user,unexpected events can make it impossible to follow the original plan.For example, in a training exercise trainees may be busy learning newskills and may not be able to keep up with the planned pace of events.Also, equipment can fail, networks can malfunction, and simulators maynot all have the same understanding of ground truth. As a result, thereis a need to monitor progress against planned training objectiveconditions during the exercise, and to offer advice and suggestions tothe instructors when things do not go according to plan.

Because the training objectives and preconditions have been defined asmathematical constraints, not only can they be used to schedule theobjectives, but these constraints can be used in the monitoring of theexecution of the user through a training scenario by populating theconstraints with actual values of the users and their actions. This isparticularly helpful if the training is conducted with the use of asoftware program because the monitoring may be integrated into thesoftware training program.

FIG. 10 shows an example of the PRESTO Monitoring Tool. By design, italigns with the tools that capture training objectives and preconditionssuch as the PRESTO Capture Tool shown in FIG. 11. The monitoring toolcan also identify conditions for emergent training objectives, that is,conditions that were not scheduled but that, should they occur,represent valuable teaching moments. When these occur, the instructorcan be notified and is given an opportunity to take action that would beinstructive for the trainees.

This example PRESTO embodiment also provides some custom 3Dvisualizations inside the 3D map view to depict training objectivecondition satisfaction monitoring. These are powered by the ground truthfacts inside the CP processing engine. An external piece of logic isloaded (a ‘visualization’) that describes how the condition is to bevisualized. For example, the WEZ (Weapon Engagement Zone) visualizationuses a distance condition check that looks like: (1) receive updatedinformation (if any) about entities in the world, (2) check rangebetween the entities specified by the condition and (3) if within range,send a “RangeEvent” EventFact to be visualized.

This EventFact is a data element. It describes the entities involved,whether they were in or out of range, and the numeric values involved.However, it has no data that says how that EventFact should bevisualized. To do that we create a “visualizer” that can visualize theseEventFacts:

-   -   public interface IFactVisualizer    -   {    -   void visualizeEvent(EventFact f);    -   }

In this embodiment, the visualizer is connected to a 3D scene, but itneed not be. It could be a 2D visualizer or it could write some data toa log file instead. This example visualizer interprets the RangeEventand draws and colors a transparent 3D sphere at the appropriate distancebetween the two entities. Using this design it is easy to add newvisualizations and even entirely new visualizers. FIG. 12 shows asequence diagram showing the flow of data through the system thatupdates the visualization.

It is understood that although the above description can apply totraining objectives and training scenarios, similar tools can be usedto: monitor scenarios for conditions that recreate a lessons-learnedscenario, monitor scenarios for conditions based on training objectives,monitor scenarios for conditions based on key computer game events andmonitor scenarios for conditions based on story plot points.

One Embodiment of Method to Capture the Conditions for TrainingObjectives:

The explicit capture of the conditions necessary for training objectivesis a step not normally taken when planning training scenarios, thoughthe training objectives do often play an implicit role. FIG. 11 shows anexample screen from a Training Objective Capture Tool. Here, the rangeof the three-dimensional angle between two aircraft is being specifiedas a precondition. This application, in general, allows thespecification of ranges of spatial, temporal, and other conditions (andcombinations of these, like velocities), and it allows the specificationof the scenario resources that will be required. It also allowsinstructors to specify the resources that will be available in thescenario, such as human-in-the-loop simulators, semi-automated forces,and the like.

It is understood that although the above description can apply totraining objectives for training scenarios, similar tools can be usedto: capture specific conditions to bring about during the scenario basedon training objectives, capture specific conditions to bring aboutduring the scenario based on key computer game events, capture specificconditions to bring about during the scenario based on story plot pointsand capture specific conditions to bring about to recreate alessons-learned scenario.

Other Application Areas:

The scenario defining and monitoring system such as PRESTO can beapplied in a variety of areas. For example, scenarios for computer gamescan be structured this way, providing interactive freedom for theplayers—but instead of training objective conditions, the tools wouldspecify key gaming events. Similarly, interactive fiction can benefitfrom this approach. Authors could specify conditions for key plotpoints, and let the players as well as the automated characters dowhatever they want as long as they meet the conditions of the key plotpoints. This would considerably reduce the need for plot branching, witha concomitant increase in the potential for plot complexity.

It is also understood that the systems and methods described can be usedby specifically programmed computer based learning systems where thetraining objectives or conditions may be a learning task or event withina broad educational subject area.

One Embodiment of a Scenario Defining System:

The various method embodiments of the invention will be generallyimplemented by a computer executing a sequence of program instructionsfor carrying out the steps of the methods, assuming all required datafor processing is accessible to the computer, which sequence of programinstructions may be embodied in a computer program product comprisingmedia storing the program instructions. A computer-based system by whichthe methods of the present invention may be carried out include acomputer system having a processing unit, which houses a processor,memory and other systems components that implement a general purposeprocessing system or computer that may execute a computer programproduct comprising media, for example a compact storage medium such as acompact disc, which may be read by processing unit through disc drive,or any means known to the skilled artisan for providing the computerprogram product to the general purpose processing system for executionthereby.

The program product may also be stored on hard disk drives withinprocessing unit or may be located on a remote system such as a server,coupled to processing unit, via a network interface, such as an Ethernetinterface. A monitor, mouse and keyboard are coupled to processing unit,to provide user interaction. A scanner and printer are provided fordocument input and output. The printer can be coupled to processing unitvia a network connection, but may also be coupled directly to theprocessing unit. The scanner can be coupled to the processing unitdirectly, but it should be understood that peripherals may be networkcoupled or direct coupled without affecting the ability of workstationcomputer to perform the method of the invention.

As will be readily apparent to those skilled in the art, the presentinvention can be realized in hardware, software, or a combination ofhardware and software. Any kind of computer/server system(s), or otherapparatus adapted for carrying out the methods described herein, issuited. A typical combination of hardware and software could be ageneral-purpose computer system with a computer program that, whenloaded and executed, carries out the respective methods describedherein. Alternatively, a specific use computer, containing specializedhardware for carrying out one or more of the functional tasks of theinvention, could be utilized.

One embodiment of a computer system that can be used for the operationsdescribed in association with any of the methods described hereinincludes a processor, a memory, a storage device, and an input/outputdevice. Each of the components are interconnected using a system bus.The processor is capable of receiving information and processinginstructions for execution within the system. The processor is capableof processing instructions stored in the memory or on the storage deviceto display information for a user interface on the input/output device.The memory provides a means to store information within the system. Insome implementations, the memory is a computer-readable storage medium.In one implementation, the memory is a volatile memory unit. In anotherimplementation, the memory is a non-volatile memory unit. The storagedevice is capable of providing mass storage for the system. In someimplementation, the storage device is a computer-readable storagemedium. In various different implementations, the storage device may bea floppy disk device, a hard disk device, an optical disk device, or atape device. The input/output device provides a means to receiveinformation and means for input/output operations for the system, suchas receiving data and may be in communication with a user interface. Inone implementation, the input/output device includes a keyboard and/orpointing device. In another implementation, the input/output deviceincludes a display unit for displaying graphical user interfaces.

The present invention, or aspects of the invention, can also be embodiedin a computer program product, which comprises all the respectivefeatures enabling the implementation of the methods described herein,and which—when loaded in a computer system—is able to carry out thesemethods. Computer program, software program, program, or software, inthe present context mean any expression, in any language, code ornotation, of a set of instructions intended to cause a system having aninformation processing capability to perform a particular functioneither directly or after either or both of the following: conversion toanother language, code or notation; and/or reproduction in a differentmaterial form.

For illustration purposes and not for limitation, one embodiment of theprogram instructions to implement the methods of this invention isdescribed below.

FIG. 19 shows the high level structural view of a PRESTO system as oneembodiment. At the top level, PRESTO has four major components. Thesecomponents are: Condition Authoring Component—used to define trainingobjectives and other information PRESTO needs to function; StaticAnalysis Component—used during training planning and scenario design tosupport instructors and scenario designers; PRESTO Core—core functionsof PRESTO that are shared between static and dynamic components ofPRESTO; and Dynamic Analysis Component—used during training exercises tosupport instructors and SAF operators to increase training opportunities

The top part of FIG. 19, 1920, shows PRESTO supporting the staticanalysis of a scenario of conditions. In static analysis, or trainingplanning-time use, the scenarios are either existing scenarios or PRESTOenhanced scenarios which are in the planning or implementation stage orhave been modified. PRESTO is used to plan and analyze the scenario todetermine if it is feasible. PRESTO does this by verifying thatfundamental conditions in the scenario meet the required conditions. Forexample, a single helicopter could not participate in two differenttraining objectives at the same time, or be in two different locationsat the same time. Other factors such as weather and time of day couldalso be considered in the feasibility determination. The results of thescenario feasibility analysis would provide information to the scenariodesigner so they could correct any issues before the scenario goes tothe simulation environment for actual training.

The lower part of FIG. 19, 1930, shows the use of PRESTO in a dynamicsituation or training run-time use situation. In this case PRESTO isused by instructors and SAF (semi-automated forces) operators to analyzereal-time data from a simulation environment and advise them on eitherplanning additional training objectives or satisfying conditions fortraining objectives to be worked on. Here the analysis is done based onactual conditions in the training exercise and PRESTO responds to thebehaviors of the entities in the simulation environment. For example, ifa trainee has a training objective of deploying a missile using buddylasing, PRESTO can monitor the conditions in the simulation environmentto verify they are correct for the buddy lasing missile deployment. Theresults of the dynamic analysis will be various visualizations hosted bythe other systems used by SAF operators and instructors.

The center box in FIG. 19, 1940, shows the PRESTO Core. This includesthe means to receive HPML which provides pedagogical information aboutthe scenario and the constraints that describe the pre-conditions neededfor the training objectives. The PRESTO Service is the core capabilityof PRESTO which is used in both the static and dynamic use cases. Onecomponent of the PRESTO Core will be the Constraint Processing Engine.This is the part of PRESTO that will be able to take in a set of valuesabout the entities in the scenario either static or dynamic (such aslocation, speed) from the simulation or from the scenario definition anddetermine if they satisfy the constraints defined for the trainingobjectives.

In one embodiment, the PRESTO Core comprises three components: thePRESTO Controller; the Scenario Analysis and ECLiPSe. The PRESTOController manages the workings of PRESTO. Its main function is toreceive requests from other components, to process them and to send theresults to the visualization components. The Scenario Analysis componentdoes the various scenario analysis functions of PRESTO. This includeschecking a scenario for feasibility, monitoring training objectivecondition satisfaction and searching for emergent training objectives.The ECLiPSe component is the constraint programming environment whichsolves the constraint problems generated by PRESTO.

The next set of architecture diagrams are Uniform Modeling Language(UML) component diagrams that show the structure of both the scenariotraining planning-time use of PRESTO and the training run-time use ofPRESTO.

FIG. 21 shows a component diagram for the scenario planning-time use ofPRESTO. This diagram shows the high level packages of components thatmake up the PRESTO system. During scenario planning-time use, PRESTO hasthe following major packages of functionality: Training ObjectiveAuthoring GUI—this is the graphical user interface (GUI) that providesone means to provide information to define training objectives and theirconditions (such as resources and timing conditions); Training ObjectiveLibrary—this is the repository of defined training objectives that canbe re-used in multiple training scenarios (HPML is used to transfertraining objective data to and from this repository); InstructorGUI—this component is used by the training instructor to specify the setof training objectives for each trainee (the training objectives foreach trainee also have priorities associated with them); Curricula—thiscomponent is a repository for the lists of training objectives that havebeen assigned to trainees; Scenario Designer GUI—this graphical userinterface is used by the scenario designer to run feasibility analysisof the scenario, to view the visualizations and to make changes to thescenario training data to fix problems; Scenario Analysis—this componentreceives information and provides the scenario analysis functions ofPRESTO such as determining feasibility of a scenario and optimizingtraining objectives; Time Visualization—this component provides avisualization of the time aspects of the scenario (this component willgenerate a GANTT-like chart of the schedule of training objectives foreach trainee); Geo/Spatial Visualization—this component will showgeometric and spatial conditions of the training objective in thecontext of the scenario being designed; and Monitoring Panel—thiscomponent will show messages to the user (for example, when checking thefeasibility of a scenario, PRESTO might determine that there are notenough resources to satisfy the conditions for a buddy lase trainingobjective and one of the ways the user will be informed of this is viathe monitoring panel).

FIG. 22 shows a component diagram of the PRESTO system as configuredduring training run-time use. The PRESTO system shares a number ofcomponents between scenario planning-time and training run-time. Theadditional components and the different features of the components ofFIG. 21 are described here: Instructor GUI—through this component theinstructor can monitor the runtime environment and visualizations andcan add or change the training objectives for a trainee based on currentconditions; PRESTO Control GUI—this component would be used by eitherthe instructor or the SAF operator to invoke various functions of PRESTOsuch as to do scenario re-scheduling or visualizations; SimulatorRuntime Environment—during training run-time, this component receivesreal-time data from the simulation environment and passes it to thePRESTO Core (the PRESTO Core uses this information to do real-timemonitoring of conditions and to generate visualizations).

One capability of some embodiments of the PRESTO system has to do withthe size of the 3D world that can be used with the system. One set ofsuitable topographical data is ETOPO which comes from NOAA (NationalOceanographic and Atmospheric Administration). ETOPO2 provides58,320,000 individual elevation values. At first this does not seem verylarge until we start attaching some data to each elevation point, sayfor example, a modest 1-32 bit elevation value (4 bytes) and 1-32 colorvalue (4 bytes). With just this modest amount of additional data, wemultiply our base value (58 million) by 8 and divide by a megabyte(1,048,576 bytes) to get a base memory value of 445 megabytes. Thiscould be a significant drain on memory resources, just dedicated toterrain data. A data set with twice the resolution, ETOPO1 (21600 by10800) provides 233,280,000 elevation points and results in a memoryallocation of 1.7 gigabytes. This can cause a problem when scaling thesystem. One approach to managing this data is to take advantage of thefact that the user is only viewing a small part of the world at anygiven time. To handle this, we broke the monolithic terrain mesh up intoterrain pages. Pages are loaded dynamically as they come into view, sowe are only using memory and processing time to handle the pages that weare using at the time. This greatly reduces the memory requirement ofthe 3D terrain data part of PRESTO. Further optimization is contemplatedwhere the user selects a given part of the world where a scenario istaking place and smaller sub-grids would be loaded instead of the wholeworld.

An Operational Example of One Embodiment of the Methods:

In operation, the methods to define training scenarios disclosedgenerally comprise the steps of defining at least one condition for atleast one training objective where the condition is represented by atleast one constraint and scheduling the conditions into a trainingscenario. In a preferred embodiment, multiple objectives, multipleconditions and multiple constraints are used to define or monitor thescenario.

Described below is one embodiment of a detailed scenario to provide datafor the PRESTO system described above. FIG. 13 shows the examplescenario having the following sequence of activities: (1) three MH-60Rhelicopters (helos) take off from ships: (2) they rendezvous at a givenlocation and fly in formation towards an area of operation; (3) once inthe area of operation, the MH-60R helos use two on-board sensors toclassify contacts in the area of operation; (4) the lead helocommunicates with MOM; and (5) two helos do a buddy lase and deploymissile training activity.

The example scenario replicates the basic scenario described here threetimes for a total of nine MH-60R helos. The scenario consists of severalentities in a virtual world that has both land and sea components. Thescenario can present opportunities for training training objectives,each having conditions that generally describe attributes of theentities and components (location, altitude, state of their radar),relationships among the entities (spacing, relative velocities), orother aspects of the world (a high sea state, cloudy weather). Theentities are considered resources for the training objectives wheneverone or more is a required part of the associated conditions. Theentities themselves can satisfy conditions of the training objective orthey can provide constraints or values to populate slots/unknowns ofconstraints. For example, other helos for a trainee to fly in formationwith are considered resources for the fly-in-formation trainingobjective. Multiple training objectives may in some cases requireoverlapping multiple resources, though in other cases a resource must bededicated to a single training objective at a time. Table 1 shows theentities contained in the example scenario.

TABLE 1 Example Scenario Entities Entity Type Entities MH-60R Three setsof three helicopters. These can represent either trainees or SAFsTargets Target ships for sensor, lasing and missile training objectivesNeutrals Neutral ships for clutter

Table 2 shows the example training objectives within this scenario. Eachtraining objective has constraints such as resource conditions andtiming conditions. For example, the “Fly in 3 ship formation” trainingobjective requires a total of 3 helos. The group of 3 helos that canparticipate in that training objective must have one helo in the leadrole and 2 other helos (not in the lead role).

TABLE 2 Example Training Objectives in Scenario Training Objective CodeResource Conditions Timing Conditions Fly in 3 ship ff 1 lead helo andNone formation 2 other helos Sensor 1 S1 At least one target None Sensor2 S2 At least one target None Classify contacts cc None After Sensor 1and Sensor 2 Communicate cm None After Classify with MOM contacts Buddylase bl At least one target, After Communicate a helo to deploy the withMOM missile Deploy missile dm At least one target, Concurrent with ahelo to lase Buddy lase, after the target Buddy lase begin, and beforeBuddy lase end.

Training objective timing conditions specify any scheduling constraintsthat exist between training objectives. For example the scheduling ofthe Deploy missile training objective must happen after the start of theBuddy lase training objective. The Buddy lase training objective mustcontinue until after the Deploy mission training objective is complete.

To further provide values for the example use case scenario, trainingpersonnel indicate the training objectives for each trainee. Thesetrainee training objectives identify some conditions to be used for thistrainee which can be put into a scenario. When multiple trainees areinvolved, these conditions are combined for a set of scenario trainingobjectives. Table 3 shows the list of training objectives that wasassigned to each of the nine trainees in this example. Each trainingobjective assigned to a trainee can be given a priority and a role. Thisdata was used to analyze and generate a schedule for the scenario.

TABLE 3 Example Scenario, Trainee Training Objectives Helo Trainee IDTraining Objective Priority aS1-1 Helo1 ff-lead 1 ff-other 5 s1 2 s2 1sf 1 cc 1 cm 1 bl 1 dm 1 fh 1 aS1-2 Helo2 ff-lead 3 Ff 1 s1 3 s2 3 sf 2cc 5 aS1-3 Helo3 ff-other 3 s1 5 s2 5 sf 5 cc 5 aS2-1 Helo4 ff-lead 2 s12 s2 2 sf 2 cc 2 aS2-2 Helo5 ff-lead 2 bl 5 dm 5 fh 5 aS2-3 Helo6 s1 1s2 3 sf 7 cc 5 cm 2 bl 1 dm 8 fh 8 aS3-1 Helo7 ff-other 1 s1 1 aS3-2Helo8 s1 5 s2 5 sf 5 cc 5 aS3-3 Helo9 s1 8 s2 8 sf 8 cc 8

For each of the training objectives (e.g. ff=Fly in 3 Ship Formationfrom Table 2), the conditions are defined by constraints which are thenrepresented mathematically. The conditions and constraints of thisembodiment follow the training objective template of FIG. 7. An exampleof this is shown below.

TABLE 4 Example of Training Objective (TO) Conditions, Constraints andMathematical Representation for Training Objective Fly in 3 ShipFormation Mathematical Representation Condition Type Constraint for Flyin 3 Ship Formation Geo/spatial Distance Distance (Lead-Helo, Conditionsbetween Helo-2) < D, TO entities Distance(Lead-Helo, Helo 3) < D,Context Weather 2 < Wind Speed < 15 Conditions conditions ResourceResource 1—Lead role Helo, Conditions Types 2—Other role Helos TimingMust follow Must follow Rendezvous Conditions TO

To complete this scenario training objective, which is combined withother training objectives to create a scenario, other constraints andrepresentations of the conditions must be identified, such as trainingobjective slots in this case. These training objective slots representvariables that will be given values during planning-time or run-timeuse. The training objective slots in this embodiment are:

TABLE 5 Example of Identifying Values for Conditions with SlotsMathematical Condition Type Constraint Representation Resource SlotsLead Role, Helo1, Other Role 1 Helo7, Other Role 2 Helo3 Geo/spatialSlots Location of Lead Role, X₁, Y₁, Z₁, Location of Other Role 1 X₂,Y₂, Z₂, Location of Other Role 2 X₃, Y₃, Z₃ Timing Slots Start time, T₁,End time T₂

The variables in the mathematical representations in Tables 4 and 5 aregiven values by entities, resources, and other information providedeither from humans (e.g. instructors) making decisions about whichtrainees will be addressing which training objectives during thescenario, by other available information about the scenario and scenarioplan, such as the availability of virtual simulators or constructivelysimulated entities in a given simulated region or automatically througha data source such as HPML. In some cases, both human input andcomputations by the constraint programming system provide values for theentity, resource, and other slots. For example, when instructors decidethat a certain trainee needs to address the fly-in-formation trainingobjective (for example, in the Lead Role), the constraint programmingenvironment will identify feasible other helos to fill in the otherentity slots for that training objective (for example, in the Other Role1 and Other Role 2). Although not a part of the embodiment describedhere, it is certainly possible that future embodiments will defer thefilling of all slots until after they have been scheduled. This wouldallow instructors to decide, for example, to schedule two groups ofthree trainees for the fly-in-formation training objective whiledeferring until later who those trainees will be and which entities theywill be flying in the scenario.

Similarly, the above process of mathematically representing theconditions and constraints for the “Fly in 3 Ship Formation” trainingobjective can be done for the other training objectives such as the“Sensor 1” training objective of Table 2.

With the mathematical representation of the above constraints thescenario can then be modeled and an optimal solution can be found withconstraint programming and/or constraint based local search. Many kindsof constraint based local search could be effective, depending on thespecific problem—tabu search, variable neighborhood search, andsimulated annealing, for example. In the present embodiment, thescheduler used a version of finite horizon scheduling, useful in arapidly-evolving environment because it doesn't attempt to schedule theentire scenario at once, coupled with simulated annealing once afeasible solution has been found.

The constraint programming engine then assembles a constraint for theobjective function (typically using the priorities on the objectivefunction, but other objective functions may be specified by the user)and solves the resulting problem optimally.

In the embodiment described here, constraint-based scheduling isaccomplished by decomposition: a main model takes care of assigning theresources to the different training objectives, whereas the sub-model(a.k.a. slave problem) tries to find the best schedule considering thegiven assignment while satisfying temporal, spatial, and resourceconstraints. FIG. 20 shows an embodiment of this decomposition process2000 graphically: the assignment model 2041 feeds possible assignmentsto the scheduling model 2043. The scheduling model attempts to improveon its initial solution. If it finds one at 2045, this is the “NewBound” branch 2048 in the diagram. If it does not find an initialsolution at 2045, it asks the assignment model for a new set ofassignments at 2046. Several assignments are provided to the sub-problemto quickly explore a fairly large yet promising portion of the searchspace. To avoid unnecessary work, the embodiment forces the slaveproblem to find solutions with a better objective value with respect tothe incumbent solution, i.e., the best solution found so far. The bestsolution found at 2047 is used to define the solution 2049.

In the constraint programming engine, a structure for shared data storesthe data structures that are useful for both the assignment and thescheduling model. In particular, it maintains two maps (both arebijections)—from trainer IDs to integer (and vice versa) and fromtrainee IDs to integer (and vice versa). The integers are used in theconstraint programming variables whereas the IDs are used forpresentation purposes and to retrieve the relevant data used forscheduling. The shared data structure also precomputes the distancesbetween all the entities (the distance matrix) for the assignment model.Last but not least, starting from the set of training objectives, itcreates the set of all training objective constraints in a form usefulto the constraint programming environment.

A software object describing the assignments of resources to trainingobjectives, the assignment model, is used by the constraint programmingmodel. Each training objective has an associated structure describingthe constraints that define the resources needed. For each resourcerequest, the model associates a decision variable whose domain rangesexactly over all the possible alternative resources in the resourcerequest. At this point, the constraint programming model does notpresent any other constraints in the current solution, but rather usesthe assignments to continue the search.

The software object describing the assignments has a method thatimplements a search heuristic that greedily tries to assign a trainee toa training objective resource request based on its average proximity tothe initial positions of the trainees involved in the trainingobjective. Each solution to this model is presented to the slave problemdescribed next. The search may be bounded by a time limit, by a maximumnumber of fails or by a maximum number of solutions found. In tests, theheuristic performed quite well, giving good assignments of resources tothe resource slots in the training objectives early on in the search.

The scheduling model is an object that fixes events in time whilesatisfying temporal, resource, and spatial constraints, to finallyproduce a schedule for the trainees. It starts by creating the set ofunary resources to represent the trainees. Then it creates a resource(either unary or discrete) for each other resource involved in thescheduling model. Next it creates for each trainee a set of events thatrepresents the tasks to be performed by the trainee. Finally, it definesfor each trainee the transition times for every activity pair: thesetransition times represent the space constraints, i.e., the time spentto fly from one location to another. The scheduling object has a methodthat implements the search strategy that ranks the trainee resources andsets the times of the events.

In the constraint programming engine, the object describing theconstraints associated with a training objective is responsible forthree tasks. First, it defines the resource requirements for thespecific training objective. In particular, it retrieves the set ofresources instances whose descriptions correspond to the required ones.The set of preselected resources in the assignment object is used as astarting place. Second, it creates the set of events that comprises thetraining objective. Note that a single training objective possiblyinvolves several events and may employ more than one trainee. Andthirdly, it posts the resource constraints and their associated temporalconstraints for the resources needed by the training objective. Theresources are considered loaded during the whole duration of thetraining objective; therefore constraints associated with a trainingobjective have a lifetime that spans its whole duration.

The result of this modeling is a schedule of activities for each traineeand each resource (entity) during the scenario. An example of schedule,or scenario, is shown in FIG. 8. Because the constraints that weresolved to come up with the schedule were in general ranges of locations,times, and so on, it will be rare that the schedule indicates exactplaces and times for events. Instead, it will indicate ranges of times,locations, and often other decision variables. These can be presented tothe instructor as a Gantt chart that indicates earliest and latest starttimes for each event.

It should be noted that in some embodiments, a scenario planning-timeresult can be obtained without having to identify specific entities orresources.

During actual training time, when the scenario is being executed, theschedule is monitored to identify for the instructor which “eventwindows” in the schedule are now open. When the constraints on thatevent window are partially but not fully satisfied—typically because thetiming is acceptable but spatial relationships or other conditions arenot—the user has access to understanding the constraints that must bemet in order for the trainee to address the intended training objective.

The system can also notify the instructor when the opportunity toaddress a training objective has passed—that is, when the current timein the simulation is later than the latest time in the Gantt chart. Whenthis has happened enough times (or at any other time the instructorchooses), the instructor may request that the constraint programmingenvironment reschedule the remaining training objectives. In doing so,it uses a process identical to the one described above except that ituses the remaining unaddressed set of training objectives and theirconstraints, the current set of resources and their locations, and anyother relevant current data as a starting place.

Example Use Cases:

The use cases contained in this section show the users of the PRESTOsystem and the primary functions they can accomplish using oneembodiment of PRESTO. The following users of PRESTO have beenidentified: (1) Training Objective Author who defines trainingobjectives and the training objective pre-conditions, (2) ScenarioDesigner who creates training scenarios and analyzes them to increasetraining opportunities, (3) Trainer who identifies the trainingobjectives that trainees should work on both pre-training and during thetraining exercise and (4) SAF Operator who monitors training activitiesduring the training exercise. PRESTO provides them with information sothey can promote the satisfaction of training objective pre-conditionsto increase training opportunities.

Training Objective Author: The training objective author definestraining objectives and their associated conditions. The use cases forthe training objective author are shown in FIG. 14. The trainingobjective author defines and edits the set of pre-conditions necessaryfor a training objective to occur. These conditions are made up ofgeospatial, timing and other conditions. Geospatial conditions includedistances, ranges and geometrical configurations (such as formations).Geospatial conditions are created using a graphical user interface. Thetraining objective author also defines timing conditions. Theseconditions describe the time dependencies between training objectives.Examples of these are TO A must precede TO B or TO C must occurconcurrent with TO D. The training objective author can also define andedit other training objective conditions such as weather and time ofday. These conditions are described using a table of values such asweather: wind greater than 10 kts and time of day: dawn.

The Training Objective Author user has the following top level usecases:

-   -   Define Training Objective—create training objectives including        the resources the training objective needs (such as a three ship        fly in formation training objective needs three helos), timing        conditions (such as sensor 1 and sensor 2 must come before        sensor fusion), geometric conditions (such as the positioning of        the target, the lasing and missile helo in a buddy lasing        activity) and other conditions (such as sea state, weather and        time of day)    -   Edit Training Objectives—edit the resources, timing dependencies        and other pre-conditions of training objectives    -   Manage Training Objective Library—Once a training objective is        defined, it can be stored in a library for reuse. These training        objectives can be applied to many trainees across multiple        training scenarios.

Scenario Designer: The scenario designer role has two major use cases.The first one involves checking the designed scenario for feasibility(shown in FIG. 15). The second major use case is enhancing the scenarioby identifying additional training opportunities in a scenario (shown inFIG. 16). The scenario designer uses PRESTO to help verify that thescenario is feasible with respect to timing conditions and resources.PRESTO uses the MSEL (Master Scenario Event List) as the initial sourceof the scenario description. Then the conditions for the trainingobjectives represented in the scenario are checked. These include timingconditions and resources needs. For example, if TO B requires TO A toprecede it, then if this is not the case in the scenario, then thescenario designer is notified of a feasibility violation. An example ofresource needs includes checking to make sure that if a trainingobjective for trainee A needs two other aircraft, that those otheraircraft are available at the time trainee A is engaged in that trainingobjective. Finally, the feasibility violation can be displayed to thescenario designer using an annotated MSEL which is color coded toindicate the type of error and the scenario designer can click on theMSEL to drill down into the details of the violation.

The Scenario Designer has the following top level use cases:

-   -   Check Feasibility—check the feasibility of the scenario by        verifying that scheduled training objectives meet all of their        conditions (such as resource requirements and sequence        dependencies). The scenario designer does this by importing a        scenario, running the PRESTO feasibility analysis function and        checking the results. This use case is shown in FIG. 15.    -   Enhance Scenario—identify areas to increase the training value        of the scenario. This includes identifying times where        additional training objectives could be added for trainees. This        use case is shown in FIG. 16.

FIG. 16 shows the use case for the scenario designer enhancing the MSELdescribed scenario by identifying additional training objectives thatcan be inserted into the MSEL without changing any of the existing MSELevents. In this case the scenario designer uses PRESTO to analyze theMSEL described scenario. PRESTO will look for “free time” periods in thescenario where trainees could take on additional training activities.The PRESTO scheduling model will then attempt to schedule traineeassociated training objectives into these periods. The schedule willtake into account the training objective conditions and resource needsas well as not adversely impacting the previously scheduled trainingactivities for others in the MSEL. The results of this analysis will bedisplayed to the scenario designer via an annotated MSEL (with colorcoding as described above) and with a GANTT chart type schedule display.

Trainer: The trainer use cases are shown in FIG. 17. The trainer isinvolved in both the scenario design time and the exercise time. Duringthe scenario design time, the trainer defines the set of trainingobjectives each of the trainees should be working on. During thetraining exercise, the trainer uses PRESTO to enhance the trainingopportunities for trainees in two ways. First, PRESTO can identifyemergent training objectives which the trainer can approve and which canbe carried out by the SAF operator and the trainees. Second, the trainercan enhance dynamic parts of the scenario MSEL. In this case, a sectionof the MSEL (referred to in the diagram as a vignette) has beendynamically selected based on the trainer or training scenarioparticipant (for example a high ranking officer). This vignettespecifies training activities for a number of trainees but might notconsider all of the training opportunities for all participants. Similarto the analyze MSEL for training objective opportunities use case forthe scenario designer, here the trainer can look for additional trainingopportunities in real time while the training exercise is in process.

The Trainer has the following top level use cases:

-   -   Identify Trainees—identify all of the trainees which will        participate in the training exercise    -   Set Curricula For Trainee—define the set of training objectives        that the trainee should make progress on. Each training        objective has a priority level for the associated trainee.    -   Monitor For Emergent Training Opportunities—the instructor can        monitor the training exercise as it is progressing and identify        additional opportunities for training. PRESTO does this by        supporting the satisfaction of training objective pre-conditions        and by scheduling additional training objectives into trainee        schedules when they meet all required conditions.    -   Dynamic Scenarios—the trainer can incorporate new scenario        fragments (called vignettes) into the running scenario to adapt        to training needs and conditions as they happen in the scenario.        The trainer can analyze the newly added vignette and look for        additional training opportunities for all of the participating        trainees.

SAF Operator: The last role in this embodiment is the SAF (SemiAutomated Forces) operator. The use cases for the SAF operator are shownin FIG. 18. The SAF operator monitors conditions for scheduled trainingobjectives during the exercise. If the conditions are in jeopardy of notbeing satisfied, the SAF operator can intervene to attempt to satisfythe conditions. The SAF operator has several visualizations(timeline/GANTT, 3D map and entities) to monitor and assess the trainingobjective conditions.

The SAF Operator has the following top level use cases:

-   -   Control SAF—this use case is provided by the SAF operator        station.    -   Monitor Training Objective Pre Conditions—the SAF operator uses        PRESTO to monitor the pre-conditions of training objectives        during the training exercise. The SAF operator uses the        pre-condition information to help them decide how to manipulate        the exercise SAF entities to help satisfy additional training        objective pre-conditions. For example, if a trainee is scheduled        to work on a sensor training objective which requires a target        within a given distance, PRESTO can help the SAF operator        monitor this pre-condition to determine if it will be satisfied.        If the pre-conditions might not be satisfied, PRESTO can help        the SAF operator determine where a target can be moved so that        it will satisfy that pre-condition.

Although this invention has been described in the above forms with acertain degree of particularity, it is understood that the foregoing isconsidered as illustrative only of the principles of the invention.Further, since numerous modifications and changes will readily occur tothose skilled in the art, it is not desired to limit the invention tothe exact construction and operation shown and described, andaccordingly, all suitable modifications and equivalents may be resortedto, falling within the scope of the invention which is defined in theclaims and their equivalents.

We claim:
 1. A computer based method of monitoring a user presented ascenario of conditions in a computer based simulator, said methodcomprising: presenting a scenario of conditions to a user with a userinterface of a computer based simulator; the scenario of conditionscomprising a plurality of conditions for at least one objective; the atleast one objective is a training objective for the user of a computerbased simulator and the plurality of conditions comprises a plurality oftraining events to be communicated by the computer based simulator tothe user; at least one of the plurality of conditions being representedby at least one constraint; the at least one constraint comprises amathematical expression having at least one variable representing aplurality of values of the at least one constraint whereby the at leastone of the plurality of conditions can be represented by the pluralityof values of the constraint; the mathematical expression implemented ina computer program; monitoring, with one or more processor, an executionof the user of the at least one of the plurality of conditions of theuser interface of the computer based simulator; determining, with theone or more processor, at least one actual constraint value of the atleast one of the plurality of conditions based on the execution of theuser; whereby the actual constraint value represents a performance ofthe user in the computer based simulator; and determining, with the oneor more processor, whether the actual constraint value satisfies the atleast one of the plurality of conditions.
 2. The method of claim 1wherein the mathematical expression comprises a range of a variable ofthe at least one of the plurality of conditions.
 3. The method of claim1 wherein: the mathematical expression is a relational expressionrepresenting a variable of the at least one of the plurality ofconditions; the at least one actual constraint value comprises an actualvalue of the variable of the at least one of the plurality ofconditions; and determining, with the one or more processor, whether theactual constraint value satisfies the at least one objective comprisescomparing, with the one or more processor, the actual value of thevariable of the at least one of the plurality of conditions to a rangeof values of the variable of at least one of the plurality of conditionsand determining whether the actual value falls within the range ofvalues.
 4. The method of claim 1 wherein the objective is an educationalobjective.
 5. The method of claim 1 wherein one of the plurality ofconditions is one of a plurality of 3D geographic locations of an entityin a simulation environment.
 6. The method of claim 1 wherein thescenario of conditions is changed dynamically during the execution ofthe user based on a second execution of a second user.
 7. The method ofclaim 1 wherein the scenario of conditions is changed automaticallyduring the execution of the user based on the actual condition.
 8. Themethod of claim 7 wherein: the scenario of conditions is changed byoptimizing a plurality of constraints using constraint programming; theat least one objective comprises at least one training objective; andthe plurality of conditions comprises one condition selected from thegroup consisting of: a length of the scenario of conditions, and aquantity of the plurality of conditions.
 9. A method of defining ascenario of conditions for computer based simulator, said methodcomprising the steps of: defining a plurality of conditions and aplurality of slots for a plurality of objectives; at least one of theplurality of conditions being represented by at least one constraint;the at least one constraint comprises a mathematical expressionconfigured to model a variable of the at least one constraint; each ofthe plurality of slots defining a value of the variable; populating oneor more of the plurality of slots with the value of the variable; andautomatically scheduling the plurality of conditions to define thescenario of conditions by optimizing the at least one objective giventhe at least one constraint across the plurality of conditions.
 10. Themethod of claim 9 wherein the plurality of objectives is a plurality ofeducational objectives.
 11. The method of claim 9 wherein the step ofautomatically scheduling the plurality of conditions to define ascenario of conditions by optimizing the at least one objective giventhe at least one constraint across the plurality of conditions comprisesautomatically sequencing the plurality of conditions to define ascenario of conditions by optimizing the at least one objective usingconstraint programming.
 12. The method of claim 9 wherein automaticallysequencing the plurality of conditions to define a scenario ofconditions by optimizing the at least one objective given the at leastone constraint across the plurality of conditions comprisesautomatically scheduling the plurality of conditions to define ascenario of conditions by satisfying the at least one objective usingconstraint programming.
 13. The method of claim 9 wherein themathematical expression comprises a range of at least two of theplurality of values of the variable of the at least one constraint. 14.The method of claim 9 wherein a value of the variable is defined by anentity.
 15. The method of claim 9 wherein a value of the variable isdefined by a resource.
 16. The method of claim 9 wherein a value of thevariable is defined by a priority.
 17. A method of defining a scenarioof conditions for a computer based simulator, said method comprising thesteps of: an assignment model receiving a plurality of conditions for atleast one objective; at least one of the plurality of conditions beingrepresented by at least one constraint; the at least one constraintcomprises a mathematical expression having at least one variablerepresenting a plurality of values of the at least one constraintwhereby the at least one of the plurality of conditions can berepresented by the plurality of values of the at least one constraint;the mathematical expression implemented in a computer program to beexecuted with a processor; the assignment model communicating at leastone of the plurality of conditions to a scheduling model; the schedulingmodel scheduling, with a processor, at least one of the plurality ofconditions to define the scenario of conditions for a computer basedsimulation environment by satisfying the at least one objective giventhe at least one constraint; the computer based simulation environmentcomprises a plurality of networked computer based simulators; wherebythe scenario of conditions comprises at least one of the plurality ofconditions for each of the networked computer based simulators; and thescheduling model communicating at least one of the plurality ofconditions to the computer based simulation environment.
 18. The methodof claim 17 further comprising: receiving an actual value of the atleast one variable of the at least one constraint from one of theplurality of networked computer based simulators; whereby the actualvalue represents an actual constraint value from a user of one of theplurality of networked computer based simulators; the scheduling modelrescheduling at least one of the plurality of conditions to define anupdated scenario of conditions for the computer based simulationenvironment by satisfying the at least one objective given the at leastone constraint and the actual constraint value; and whereby the updatedscenario of conditions comprises at least one of the plurality ofconditions for each of the networked computer based simulators.